Nov 19
7
One of the takeaways from the Google Webmaster Conference was that if Google tries to access your robots.txt file is unreachable but it does exist then Google won’t crawl your site. Google said about 26% of the time GoogleBot cannot reach a robots.txt file
Read the original post:
Google & WordPress Robots.txt Handling Is Being Looked Into